Organizations with a large number of computers that run several different services typically monitor both hardware and software events for anomalies that can indicate security threats. To date, operational security assurance procedures are typically based on rules that analyze events for pre-defined patterns. For example, the rules may be run against logs of each computer. The pre-defined patterns can indicate a potential security threat which, once identified, can be addressed. This rule-based approach can fail to scale in at least two dimensions, thus rendering the approach difficult to flexibly implement. First, regarding rule-based implementations, such requires coding of the rules ahead of time, based on expert knowledge. This means that rule developers have to anticipate what is sought as a vulnerability. Rule developers may not, however, be knowledgeable of all potential vulnerabilities, thus leaving gaps in the approach. Second, during operation, the rule-based approach demands full scanning of all events, seeking for patterns in data or information that may have incomplete or incorrect data.
Additionally, conventional testing techniques for these services rely on scheduled testing or event-based testing. In scheduled testing, tests may become repetitive and result in significant over testing due to repetition of tests that pass cleanly regardless of whether or not there is a reason to suspect a different outcome, thus resulting in inefficient use of resources. In cases of conventional one-off event-triggered testing, code check-in and deployment may occur without each development team being aware in a complex or distributed service with many development teams working in parallel. Additionally, dependencies can then change underneath portions of the service, which may introduce failure states without the developers realizing anything has changed.